simChef: an intuitive framework for reliable simulation studies in R

useR! 2022 – Poster Session

James Duncan1, Tiffany Tang1, Corrine F. Elliott1, Philippe Boileau1, Bin Yu1
1University of California, Berkeley
Unpublished,

Overview

simChef is a powerful framework for designing, documenting, and deploying simulation experiments in R.

Goals:

Simulation abstractions

Experiment: a collection of related simulation scenarios.

DGP: Data-generating processes, which define the “ground truth” and flexibly generate simulated data.

Method: The main objects of study, along with baselines.

Evaluator: Produce meaningful statistics and summaries of the results.

Visualizer: Output plots, tables, R Markdown, latex, etc. to populate interactive experiment documentation.

Grammar of simulations

experiment <- create_experiment() %>%
  add_dgp(dgp1) %>%
  add_dgp(dgp2) %>%
  add_method(method1) %>%
  add_vary_across(
    dgp = dgp1,
    n = c(100, 1000, 10000)
  ) %>%
  add_vary_across(
    method = method1,
    lambda = c(0.1, 0.5, 1.0)
  )

results <- experiment %>%
  run_experiment()

Interactive documentation

Interactive R Markdown simulation documentation

Parallelization

simChef plus future R package hex logos
library(future)
n_workers <- availableCores()
plan(multisession, workers = n_workers)

results <- experiment %>%
  run_experiment(n_reps = 200,
                 save = TRUE)
Fitting empirical-fdr-comparison...
Saving fit results...
Fit results saved | time taken: 0.252786 seconds
200 reps completed (totals: 200/200) | time taken: 17.675338 minutes
==============================

Other helpful features:

Roadmap

simChef hex logo

Thank you!

Acknowledgements

We acknowledge a number of funding sources and grants that help support our research, including the Weill Neurohub, C3.ai, and the NSF.